Method and apparatus for video deinterlacing and format conversion

ABSTRACT

A method for deinterlacing a picture is disclosed. The method generally includes the steps of (A) calculating a potential sample at a location interfaced with a first field of the picture by temporal filtering, (B) evaluating a protection condition in a current region around the location after inclusion of the potential sample and (C) calculating an interpolated sample at the location by vertical spatial filtering the first field in response to the protection condition indicating a significant increase in a vertical activity within the current region due to the potential sample.

This application is a continuation of U.S. application Ser. No.10/744,729, filed Dec. 23, 2003, which is hereby incorporated byreference in its entirety.

The present invention is related to U.S. Pat. No. 7,170,561, filed Dec.4, 2003 and co-pending U.S. patent application Ser. No. 10/744,693,filed Dec. 23, 2003, which are hereby incorporated by reference in theirentirety.

FIELD OF THE INVENTION

The present invention relates to digital video formatting generally and,more particularly, to a method and an apparatus for video deinterlacingand format conversion.

BACKGROUND OF THE INVENTION

Digital images and video come in a large variety of formats. For manyapplications, converting between two or more different formats isdesirable. A high-quality low-cost method for converting the digitalsignals is very useful for such applications as: (1) convertinginterlaced NTSC video at 60 fields/second to progressive video with asimilar or larger horizontal and vertical resolution at 60 frames/secondfor display on progressive televisions, (2) performing a high-quality“zoom” function on either interlaced or progressive video and (3)increasing the horizontal and/or vertical resolution of progressive orinterlaced video or images.

Existing solutions for video deinterlacing include bob (i.e., verticalspatial filter), weave (i.e., temporal filter), VT-filter (i.e.,vertical spatial filter combined with temporal filter, commonly fixedfiltering that combines a highpass version of a previous opposite parityfield with a lowpass interpolation of a missing line from a currentfield), motion-adaptive and motion-compensated techniques and edge-basedspatial filtering. The various video techniques that are not temporal innature are applicable to image up-conversion (i.e., vertical andedge-based spatial filtering). Horizontal and edge-based spatialfiltering is used for horizontal upsampling of images or video.

Bob (i.e., vertical filtering) is known to produce temporal flickeringartifacts in video and reduced vertical detail in both images and video.In vertical filtering, odd and even lines are alternately blurred in thevideo by interpolation in a vertical direction only from adjacent lines.A resulting lack of vertical detail is particularly noticeable for sharpedges. Weave (i.e., temporal filter) is known to produce “jaggies”(i.e., interlace artifacts that are extremely objectionable for movingobjects). The VT-filtering is a fixed (i.e., non-adaptive) low-costline-based process that is cost effective to implement in silicon but isknown to produce temporal artifacts (i.e., trailing edges or “edgeghosts” from previous fields appear behind moving objects).

Motion adaptive techniques commonly make pixel-level, block-level and/orpicture-level decisions about whether to use weave or bob or a blendedcombination of weave and bob for particular pixels, blocks and/orpictures. Weave is a good option for still portions of video and a poorchoice for moving areas. Hard block-level decisions in motion adaptivetechniques can lead to objectionable blocking artifacts. However, moreadvanced motion adaptive deinterlacing techniques that combine weave andbob suffer mainly from relatively poor performance for moving video dueto all the drawbacks of bob. For stationary regions, however, theflickering artifact created by bob may be reduced.

Motion compensated techniques operate in a similar manner to motionadaptive techniques, except that rather than always using co-locatedpixels from a previous opposite parity field to replace missing pixelsin a progressive frame that is formed from the current field (i.e.,weave), motion compensated pixels are chosen from the previous oppositeparity field. An advantage of the motion compensated technique is thatgood deinterlacing is achievable for moving video that can be wellestimated. A disadvantage of the motion compensated technique is thatmotion estimation is often more expensive than any of the previouslymentioned techniques. If motion estimation fails on the video sequence(i.e., highly irregular motion, non-smooth motion fields or variouslighting effects), motion compensated techniques may be no better thanless complex methods. Furthermore, even when motion estimation issuccessful, an amount of high-frequency information that can betransferred from the previous opposite parity field to the estimate ofthe missing lines for reconstruction a progressive frame from thecurrent field depends upon a sub-pel motion between the two fields. In aworst case, objects can move by an integer number of pels plus exactlyone-half pel in the vertical direction in the temporal interval betweenthe previous field and current field. Therefore, no additionalhigh-frequency vertical information for the missing lines of the currentfield is gleaned from the previous field through the motion compensatedestimate. In practice, however, motion compensated deinterlacingincreases vertical detail while reducing flickering artifacts on a broadrange of video, such that a common drawback is simply complexity.

Edge-based spatial filtering operates on only the current field and iscapable of producing a better estimate of the pixels from the missinglines than what is possible with vertical filtering only. To a lesserextent than bob, edge-based spatial filtering also suffers from lack ofvertical detail. In particular, high frequency textures that lack edgeswill not be improved over simple bob.

SUMMARY OF THE INVENTION

The present invention concerns a method for deinterlacing a picture. Themethod generally comprises the steps of (A) calculating a potentialsample at a location interfaced with a first field of the picture bytemporal filtering, (B) evaluating a protection condition in a currentregion around the location after inclusion of the potential sample and(C) calculating an interpolated sample at the location by verticalspatial filtering the first field in response to the protectioncondition indicating a significant increase in a vertical activitywithin the current region due to the potential sample.

The objects, features and advantages of the present invention includeproviding a method and an apparatus for video deinterlacing and formatconversion that may (i) eliminate an exhaustive evaluation of allcandidate directions by exploiting a convex nature of a metric searchspace, (ii) reuse identical silicon for both intra-mode estimation andedge detection for deinterlacing, (iii) use a different number of filtertaps for vertical interpolation compared to directional interpolation,(iv) exploit homogeneity along edges that may be strong and regularenough to be detected without admitting artifacts from false detects,(v) prevent artifacts in areas containing small apertures, (vi) detectstatic horizontal edges, (vii) detect static areas of small spatialextent, (viii) reduce a severity of artifacts from false angledetections, (ix) increase a confidence level of decisions, (x) provide arobust and correct confidence estimation for an angle detection, (xi)refine a decision between angles adjacent to a best angle detected by afirst rough and cheap estimator, (xii) operate at a lower clock speed,(xiii) reduce microprocessor utilization, (xiv) reduce silicon area byre-using silicon designed for video compression for an additional taskof deinterlacing, (xv) reduce interpolation complexity for non-verticaldirectional interpolation without sacrificing quality, (xvi) permit adetection of edges to be more aggressive in areas without smallapertures thereby increasing visual performance on edges withoutincreasing artifacts from false detections, (xvii) reduce annoyingflickering artifacts, (xviii) increase quality of low-contrast andmarginally detectable edges, (ixx) increase correction detectionprobability, (xx) reduce false detection probability and/or (xxi)increase quality through decreasing a probability of detecting an angleadjacent to an optimal angle.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features and advantages of the presentinvention will be apparent from the following detailed description andthe appended claims and drawings in which:

FIG. 1 is a block diagram of an example field availability;

FIG. 2 is a flow diagram for an example implementation of adeinterlacing method in accordance with a preferred embodiment of thepresent invention;

FIG. 3 is a diagram of example fields used in a static check with anexample of a neighborhood used in the static check;

FIG. 4 is a diagram of an example picture having a stationary horizontaledge;

FIG. 5 is a diagram of example fields used in a horizontal stationaryedge check;

FIG. 6 is a diagram of example horizontal stationary edge conditions;

FIG. 7 is a diagram of an example field being deinterlaced;

FIG. 8 is a diagram of an example edge detection window;

FIG. 9 is a diagram of example search angles relative to a horizontalaxis;

FIG. 10 is a diagram for an example edge search at a 45 degree angle;

FIG. 11 is a diagram for an example edge search at a 117 degree angle;

FIG. 12 is a diagram illustrating an example condition for a protectionmechanism;

FIG. 13 is a diagram of an example small aperture detection window;

FIGS. 14 a and 14 b are diagrams illustrating an example blendingmethod;

FIG. 15 is a diagram of multiple fields used in an example static checkof two samples;

FIG. 16 is a diagram of an example horizontal stationary edge window;

FIG. 17 is a diagram of an example edge detection window;

FIG. 18 is a diagram of an example small aperture detection window; and

FIG. 19 is a flow diagram of an example iterative deinterlacing method.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention may be realized as a low-cost high qualitydeinterlacing technique. The technique may provide spatial filtering byconsidering directional information at a pixel level in a field. Lowcost may be realized through the following methods that may beoptionally applied to reduce implementation complexity. Theimplementation complexity may depend on an application platform (e.g.,fully custom hardware, hardware processor extension or full softwareimplementation).

Referring to FIG. 1, a block diagram of an example field availability isshown. A frame 100 in a picture may be generated by calculating aninterpolated field (e.g., OUT) interlaced with a current field (e.g.,CURR) of the picture. Generation of the interpolated field OUT may beperformed using one or more of (i) a first previous field (e.g.,PREV_OPP) having an opposite parity (e.g., top or bottom, odd or even)as the current field CURR, (ii) a second previous field (e.g.,PREV_SAME) having a same parity as the current field CURR, (iii) a firstnext field (e.g., NEXT_OPP) having the opposite parity as the currentfield CURR and (iv) a second next field (e.g., NEXT_SAME) having thesame parity as the current field CURR. Both previous field PREV_SAME andPREV_OPP may occur earlier in a display order and/or time than thecurrent field CURR. Both next fields NEXT_SAME and NEXT_OPP may occurlater in the display order and/or time than the current field CURR. Ingeneral, each of the fields PREV_SAME, PREV_OPP, NEXT_SAME and/orNEXT_OPP may be any field in the vicinity of the current field CURRincluding, but not limited to, adjoining and/or immediately adjacentfields. For simplicity, only the previous fields PREV_SAME and PREV_OPPmay be illustrated in some of the examples below to illustrate thepresent invention, although any one or more of the fields may be used.

Referring to FIG. 2, a flow diagram for an example implementation of adeinterlacing method 102 is shown in accordance with a preferredembodiment of the present invention. The deinterlacing method 102generally comprises a step (or block) 104, a step (or block) 106, a step(or block) 108, a step (or block) 110, a step (or block) 112, a step (orblock) 114 and a step (or block) 116. A signal (e.g., PIXEL) may bereceived at the step 104. A signal (e.g., OUTPUT_PIXEL) may be generatedby the blocks 106, 112 and 116.

The step 104 may be operational to perform a static check in a regionaround a location of the interpolated field OUT where an interpolatedsample is to be created. The region may cover N (e.g., 3) horizontal byM (e.g., 3) vertical pixels or samples (e.g., a luminance component of apixel). Other region sizes may be implemented to meet the criteria of aparticular application. If the step 104 concludes that the interpolatedsample may be in a static area of the picture (e.g., the YES branch ofstep 104), the method 102 may proceed to the step 106. If the step 104concludes that the interpolated sample may be in an area of the picturewith motion (e.g., the NO branch of step 104), an additional check maybe performed by the step 108.

The step 106 may be operational to perform temporal filtering (e.g.,weave) to generate the interpolated sample from the current field CURRand one or more of the previous fields PREV_OPP and/or PREV_SAME. In oneembodiment, the temporal filtering may combine the current field CURRand only the opposite parity previous field PREV_OPP. Other temporalfiltering methods may be implemented to meet the criteria of aparticular application.

The step 108 may be operational to detect a presence of a horizontalstationary edge in the picture (e.g., current field CURR). If thelocation of the interpolated sample is on a static side of a horizontalstationary edge (e.g., the YES branch of step 108), the temporalfiltering of the step 106 may be used to calculate the interpolatedsample. If the location of the interpolated sample is not on a staticside of a horizontal stationary edge or no horizontal stationary edgeexists in the picture proximate the location (e.g., the NO branch ofstep 108), another check is made by the step 110 for other edges in thepicture.

The step 110 may be operational to detect non-horizontal edges in thepicture (e.g., the current field CURR). If a good usable edge isdetected at or near the location of the interpolated sample (e.g., theYES branch of step 110), the method 102 may mark and edge as found andperform a directional filtering per the step 112. If no edges aredetected or all of the edges detected are unusably weak and thus markedas not found (e.g., the NO branch of step 110), vertical filtering maybe used to generate the interpolated sample.

The step 112 may be operational to calculate the interpolated sample byspatial filtering along an angle corresponding to the edge detected bythe step 110. The step 114 may be operational to calculate theinterpolated sample by vertical (spatial) filtering. The step 116 may beoperational to modify the interpolated sample generated by the step 114by blending with a co-located sample Xco or a motion compensated sampleXmc. The samples Xco and Xmc may be calculated either from (i) theprevious field PREV_OPP and the next field (NEXT_OPP) having the sameparity as the previous field PREV_OPP or (ii) the previous fieldPREV_SAME and the next field (NEXT_SAME) having the same parity as theprevious field PREV_SAME.

Referring to FIG. 3, a diagram of example fields used in a static checkis shown. The location of the interpolated sample may be illustrated bya letter X. Locations of known samples may be illustrated by the lettersA, B, C, D, E and F. The reference letters A, B, C, D, E, F and X mayalso be used to represent sample value or pixel values in mathematicalequations and boolean operations discussed below.

For the current field CURR and the previous field PREV_SAME, thelocations B and E may be positioned directly above and below thelocation X, respectively. The locations A and C may be immediately tothe left and right of the location B, respectively. The locations D andF may be immediately to the left and right of the location E,respectively. For the previous field PREV_OPP, the location X may be thesame as the location B.

The static check step 104 may include (i) any conditions that specify anapproximate equality of a set of neighboring sample values or pixels inthe current field CURR and the previous field PREV_SAME and (ii) any“protection” conditions in areas of significant vertical activity. Forthe static check step 104 to be evaluated as true (e.g., the YESbranch), both the equality condition and the protection condition shouldevaluate as true. A specific equality condition may aggregate multiplepixel-by-pixel comparisons over the set of neighboring pixels. Apreferred technique may be to compare the sample values at the locationsA, B, C, D, E and F between the current field CURR and the previousfield PREV_SAME. If PREV_SAME (A,B,C,D,E,F) are approximately equal toCURR (A,B,C,D,E,F), then the interpolated sample at the location X maybe calculated as the known sample at location B in the previous fieldPREV_OPP. Generally, samples “a” and “b” may be considered approximatelyequal (e.g., a≅b) if either or both of equations 1 or 2 are true asfollows:

|a−b|<p1  Eq. (1)

p2*|a−b|<min(a,b)  Eq. (2)

where p1 and p2 are programmable parameters with suggested values of 6and 15, respectively.

In general, the static check may be performed to avoid the interpolatedpixel changing local vertical activity dramatically. Therefore, theprotection condition may be performed as part of the static check. Theprotection condition may specify that the vertical activity after theinclusion of the inserted pixel or pixels is not much greater than thevertical activity of the neighborhood of pixels in the original field.An example protection condition may be defined per equation 3 asfollows:

|B−X|+|X−E|<p3*|A−D|+p4*|B−E|+p3*|C−F|  Eq. (3)

where p3 and p4 are programmable parameters with default values of 1 and2, respectively, X is the interpolated sample and A, B, C, D, E and Fmay be known samples from the current field CURR. Other protectionconditions may be implemented to meet the criteria of a particularapplication. The protection mechanism may impact interpolationperformance in areas of high vertical frequency but generally preventspossible artifacts in areas of high temporal frequency.

Referring to FIG. 4, a diagram of an example picture 120 having astationary horizontal edge is shown. The picture 120 may comprise afirst region 122 and a second region 124. A horizontal edge or boundary126 in the picture may separate the region 122 from the region 124. Theregion 122 may present spatially varying content of the picture. Theregion 124 may present spatially static content of the picture. Theregion 122 may be above the region 124 (as shown) or the region 124 maybe above the region 122 (not shown). The horizontal stationary edgesituation may frequently occur in sports sequences and news sequences.If not accounted for, interpolated pixels along the horizontal staticedge 126 may result in a noticeable line flickering.

Referring to FIG. 5, a diagram of example fields used in the horizontalstationary edge check are shown. The horizontal stationary edge checkmay check one or more conditions to determine if a horizontal stationaryedge is present. A first condition may specify the horizontalneighborhood pixels (e.g., G-N) to have sufficiently low variation. Asecond condition may specify a vertical variation to be much larger thana variation among the pixels G-N. Additional conditions may be includedin the check to meet the criteria of a particular application. Forexample, if the interpolated sample is located on the stationary contentside of the edge 126, the horizontal stationary edge check (step 108)and subsequent weave (step 106) may be determined by equation 4 asfollows:

If (1) max(G,D,E,F,N)<p5*min(G,D,E,F,N) is true  Eq. (4)

-   -   for both CURR and PREV_SAME, where p5 is a programmable        parameter with a suggested value of 1.2,    -   && (2)        min(|H−G|,|A−D|,|B−E|,|C−F|,|M−N|)>p6*(p7+maxdiff(G,D,E,F,N)) is        true for both CURR and PREV_SAME, where maxdiff( . . . )=max( .        . . )−min( . . . ) and where p6 and p7 are programmable        parameters with suggested values of 5 and 2 respectively,    -   && (3) PREV_SAME(G,D,E,F,N)≈CURR(G,D,E,F,N)    -   && (4) CURR(E)≈PREV_OPP(B) (e.g., protection)    -   then X=PREV_OPP(B)        Condition (1) may be true where a range of the stationary side        of the edge 126 is small. Condition (2) may be true if a minimum        gradient across the edge 126 is greater than the range of the        stationary side of the edge. Condition (3) may be true if one        side of the edge 126 is stationary (e.g., a lower edge G, D, E,        F, N as shown). Condition (4) may be true if X is on the        stationary side of the edge. If any of the conditions (1)        through (4) are false, the horizontal stationary edge check may        fail and the step 110 may check if the location of the        interpolated sample is proximate an edge of the picture.

Referring to FIG. 6, a diagram of example horizontal stationary edgeconditions are shown. Fields 130, 132 and 134 generally illustratesituations where a lower horizontal edge (e.g., G, D, E, F, N) may bestationary. Fields 136, 138 and 140 generally illustrate situationswhere an upper horizontal edge (e.g., H, A, B, C, M) may be stationary.The field 130, 134, 136 and 140 generally illustrate situations wherethe interpolated sample is located near a corner of the picture.Flickering may also be reduced through (i) applying a deflickeringfilter to the deinterlaced frames and/or (ii) performing a pixel-basedstatic check using two fields of the same parity and/or horizontalstatic edge check using two fields of the same parity and/or a staticcheck using two fields (e.g., using an absence of weave artifacts as anindicator of static regions).

Referring to FIG. 7, a diagram of an example field 150 beingdeinterlaced is shown. If the interpolated sample X is located near oron an edge 152, spatial interpolation along an orientation of the edge152 may be used instead of vertical filtering.

Referring to FIG. 8, a diagram of the example edge detection window 160is shown. The edge detection window 160 may span an number (e.g., 2 to32) of pixels/samples horizontally and a number (e.g., 2 to 8) of fieldlines vertically. The location X may be disposed at approximately acenter of the edge detection window 160.

Referring to FIG. 9, a diagram of example search angles 162 a-162 nrelative to a horizontal axis 164 are shown. The search angles 162 a-162n may be disposed in the edge detection window 160 centered on theinterpolated sample location X. In one embodiment, the search angles 162a-162 n may be angularly offset relative to the horizontal axis 164 byequations 5a or 5b as follows:

Angle=tan¹(2/a), where a=0, 1, . . . n  Eq. (5a)

Angle=180°−tan⁻¹(2/a), where a=1, . . . n  Eq. (5b)

As such, an angular separation between neighboring search angles 162a-162 n may vary as a function of the angles. For example, the angularseparation between the search angles 162 e (e.g., 90 degrees) and thesearch angle 162 f (e.g., 63 degrees) may be 27 degrees. However, anangular separation between the search angles 162 a (e.g., 166 degrees)and the search angle 162 b (e.g., 164 degrees) may be only 2 degrees. Inanother embodiment, the search angles 162 a-162 n may be angularlyseparated by a constant amount (e.g., 15 degrees).

The hierarchical angle search method for detection of angles foredge-based spatial filtering may be applied to a serial hardwareimplementation (e.g., a single hardware unit, potentially implemented asa custom hardware extension to a microprocessor) may be designed in aprogrammable way such that the method may calculate a matched filter“goodness of fit” metric for a large number of possible edge directions(e.g., 15, 30, 45, 60, 75, 90, 105, 120, 135, 150 and 165 degreeangles). A brute force implementation would utilize a single invocationof a unit implementing the method for each of the angles (e.g., 11)under consideration. However, equal performance may be achieved by firstevaluating some primary angles (e.g. 15, 45, 75, 105, 135 and 150 degreeangles), then computing the scores for the two immediately neighboringangles of the best angle from the primary angles (e.g., only 8evaluations instead of 11). The winning angle may be the overall bestscore among the evaluated angles. By using multiple levels of hierarchy,a total number of angles evaluated for each interpolation location maybe significantly reduced.

Referring to FIGS. 10 and 11, diagrams for example edge searches at a 45degree angle and a 117 degree angle are shown. The edge search may beperformed along a number of line segments 170 a-170 m. Each of the linesegments 170 a-170 m may be oriented parallel to each other and parallelto the particular search angle 162 a-162 n under consideration. The linesegments 170 a-170 m may collectively be referred to as a set.

Two diagnostic parameters (e.g., d and t) may be generated for the setat each individual search angle 162 a-162 n. The diagnostic parameter“d” may be referred to as a “first score”. The diagnostic parameter “t”may be referred to as a “second score” The diagnostic parameters for the45 degree search angle may be calculated by equations 6 and 7 asfollows:

d(45)=|A−T|+|B−G|+2*|C−D|+|M−E|+|U−F|+PEN(45)  Eq. (6)

t(45)=(|(A−B)−(T−G)|+2*|(B−C)−(G−D)|+2*|(C−M)−(D−E)|+|(M−U)−(E−F)|)*5  Eq.(7)

The diagnostic parameters for the 117 degree search angle may becalculated by equations 8 and 9 as follows:

d(117)=|S−G|+|H−D|+|A−E|+|B−F|+|C−N|+|M−V|+PEN(117)  Eq. (8)

t(117)=(|(S−H)−(G−D)|+2*|(H−A)−(D−E)|+4*|(A−B)−(E−F)|+2*|(B−C)−(F−N)|+|(C−M)−(N−V))|)*3  Eq.(9)

Shallower edges may be penalized relative to steeper angles by adirectional edge detection unit (not shown) performing the edgedetection operation in the step 110. A penalty value (e.g., PEN) may beadded to the first score d. The penalty value PEN may be a function ofthe search angle 162 a-162 n. An advantage of the penalty may be thatfalse detections of steeper angles 162 a-162 n generally result in lesssevere artifacts than false detections of shallower angles 162 a-162 n.Penalizing the shallower angles 162 a-162 n may be useful when trying torecognize even quite weak, low-contrast or marginally detectable edges.Another technique to ignore low-contrast angles is through the mechanismof reducing pixel/sample bit-depth in the edge detection window 160. Anexample of the penalty value PEN as a function of a search angle (e.g.,ANG) is generally provided in TABLE I as follows:

TABLE I ANG 63, 45, 34, 27, 22, 18, 16, 14, 90 117 135 146 153 158 162164 166 PEN 0 24 42 60 78 96 114 132 160

A directional averaging operation may be performed by the step 110 aspart of the edge orientation decision. Using a number of variables(e.g., α and β) defined by equations 10 and 11, an edge detectiondecision may be given by equation 12 as follows:

α=arg min(d(i)), iε{14°, 16°, L, 166°}  Eq. (10)

β=arg min(d(i)), iε{14°, 16°, L, 166°} and i≠α  Eq. (11)

If d(α)<Threshold_(—) d (e.g., 192)  Eq. (12)

-   -   && α and β are immediate neighbors or d(α)*p8<d(β), where p8 is        a programmable parameter with a suggested value of 2,    -   && t(α)<Threshold_t (e.g., 1200)

then X=(U+V)/2

where U and V may be the two samples along the direction αL, andThreshold_d and Threshold_t may be predetermined values.

Once the step 110 has determined (i) that one or more suitable edgeshave been detected and (ii) a best angle among one or more anglescorresponding to the one or more detected edges, the step 112 maygenerate the interpolated sample X by performing directional filteringbased on the best angle. The best angle may be associated with a bestscore among the multiple scores generated for the various search angles162 a-162 n.

Calculating a score for a particular search angle generally involvescomputing scores along each of the several parallel line segments 170a-170 m within the set for the particular search angle. The line segmentscores may then be averaged together either (i) with the same weights oneach score or (ii) with different weights. Because of the averaging, thefollowing situation may happen. A best score may be determined aftercalculating the average scores for each of the search angles. However,an actual best interpolation angle may be slightly different from the“best” angle (e.g., BA) corresponding to the best score. Therefore,additional comparisons may be performed within a small window around theangle BA. The additional comparisons may be performed among the angle BAand two immediate neighboring angles, one on each side of the angle BA.A true best angle may be used to interpolate the sample X.

A first example averaging operation using the 45 degree search angle maybe performed per equation 13 as follows:

dm=min(|B−D|,|C−E|,C−G|,M−D|,|C−D|)  Eq. (13)

if dm=|B−D|, then X=(B+D)/2

if dm=|C−E|, then X=(C+E)/2

if dm=|C−G|, then X=(C+G)/2

if dm=|M−D|, then X=(M+D)/2

if dm=|C−D|, then X=(C+D)/2

A second example averaging operation using the 117 degree search anglemay be performed per equation 14 as follows:

If |B−E|<min(|A−F|,|A−E|,|B−F|),then X=(B+E)/2  Eq. (14)

else if |A−F|<min(|B−E|,|A−E|,|B−F|), then X=(A+F)/2

else X=(A+B+E+F)/4

For vertical filtering, less blurring may be achieved through a use of alarge number of filter taps (e.g., a 4-tap filter is generally capableof retaining high vertical frequencies better than simple 2-tap linearinterpolation). However, for edge-based spatial filtering, detectabledirectional edges may be nearly homogeneous. Successfully recognizededges with a high confidence may practically benefit from edge-basedspatial filtering without a significant false detection rate tointroduce artifacts. Therefore, a larger number of filter taps mayprovide no additional benefits. Instead, a simple bilinear interpolationin a direction of the edge is generally sufficient for attaining highquality while also being an efficient and low cost method to implement.

Some systems that incorporate edge-based spatial interpolation may alsoincorporate directional predictors for intra-estimation for videocompression (e.g., 4×4 and 16×16 pel intra-prediction mode decisionestimators of the H.264 standard). When intra-prediction modeinformation is available, the edge-based spatial interpolation mayeither (i) enhance the pixel-based directional edge decision with theblock-based intra-prediction information available from an intraestimation unit or (ii) reduce silicon cost by replacing a custompixel-based edge estimator with the block-based intra-prediction modeestimator.

Confidence numbers may be used within the directional filtering of thestep 112. By way of example, consider k candidate angles (e.g., α1-αk),with respective scores d(α1) to d(αk) (e.g., the smaller a value of ascore d(i), the more probable the corresponding angle αi will be used).A confidence number (e.g., CONF) may be used as a measure of theconfidence level of the angle αi. The confidence number CONF may then becompared to a threshold. If the best angle αi has a confidence numberCONF greater than the threshold, the best angle αi may be used for thedirectional filtering. Otherwise, vertical filtering may be used tocalculate the interpolated sample. The confidence number CONF may bedetermined by equation 15 as follows:

$\begin{matrix}{{C\; O\; N\; {F( {\alpha \; i} )}} = \frac{1/{d(i)}}{\sum\limits_{j = 1}^{k}\; {1/{d(i)}}}} & {{Eq}.\mspace{14mu} (15)}\end{matrix}$

Referring to FIG. 12, a diagram illustrating an example condition for aprotection mechanism is shown. Generally, a wrong direction (e.g., angleα2) may be better than a correct direction (e.g., angle α1), possiblydue to noise or a nearby small aperture in the picture (e.g., atlocation G and/or M). Therefore, the step 110 may perform a protectioncheck after a suitable edge has been detected in the picture. If theprotection check determines that the correct angle α1 has a problem,then the method 102 may proceed with the vertical filtering step 114instead of the directional filtering step 112.

Referring to FIG. 13, a diagram of an example small aperture detectionwindow 180 is shown. The small aperture detection window 180 generallycomprises a number of columns (e.g., 3 to 13) horizontally and a numberof lines (e.g., 2 to 8) vertically. The interpolated sample location Xmay be centered in the small aperture detection window 180 between afirst line (e.g., i) and a second line (e.g., i+1) in a column (e.g.,j). In the example shown, the small aperture detection window 180 mayinclude locations H, A, B, C, M, G, D, E, F and N.

A method to detect small apertures and disable non-vertical edge-basedspatial filtering in regions containing small apertures may be providedin the step 110. In a region with a small aperture, there is generallyinsufficient evidence for which direction (or angle) the correct edgeoriented. Artifacts may result if a wrong direction (or angle) isdetermined for edge-based spatial (direction) filtering. Since shallowerdetected edges generally have a greater a potential for causingartifacts in the sample interpolations due to the false detection,vertical filtering or vertical filtering with temporal blending may beforced in the small aperture regions. An example group of steps fordetecting a small aperture in a region near the location X may be asfollows:

-   -   (1) minC=min(up[j−2], . . . , up[j+2])=up[h].    -   (2) maxL=max(up[h−1], up[h−2], up[h−3], up[h−4]).    -   (3) maxR=max(up[h+1], up[h+2], up[h+3], up[h+4]).    -   (4) Repeat steps (1), (2), and (3) for lower line.    -   (5) If maxL>>minC<<maxR is true for both upper and lower lines,        then location X is in a small aperture area.    -   (6) Repeat steps (1) to (5) with min and max exchanged and <<        and >> exchanged.

In the above steps, “>>” and “<<” stand for substantially greater andsubstantially smaller, respectively.

Referring to FIGS. 14 a and 14 b, diagrams for an example blendingmethod are shown. FIG. 14 a illustrates a group of known samples (e.g.,A, B, C and D) on consecutive odd (or even) lines (e.g., i−3, i−1, i+1and i+3) and the interpolated sample X on an even (or odd) line (e.g.,i). A parameter (e.g., i) may be calculated for the interpolated sampleX based on the known samples A, B, C and D per equation 16 as follows:

τ=(max(|A−B|,|B−C|,|C−D|))/2  Eq. (16)

A blending value (e.g., v) may be calculated per equation 17 as follows:

ν=(|Xmc−Xvf|−τ)/OFFSET  Eq. (17)

where Xmc may be a temporally estimated value for the interpolatedsample location generated by blending step 116, Xvf may be a verticallyfiltered estimated value for the interpolated sample location generatedby the vertical filtering step 114 and OFFSET may be a user programmablenumber within a predetermined range (e.g., 2 to 64).

FIG. 14 b generally illustrates the blending value v as a function ofthe parameter τ. The step 116 may generate the interpolated sample X byblending the vertical filtered value with the motion compensated valueper equation 18 as follows:

X=Xmc+ν*(Xvf−Xmc)  Eq. (18)

The parameter τ may be calculated to favor the spatially estimated valueXvf over the temporally estimated value Xmc. Two example approaches forcalculating the parameter τ may be provided by equation 19 (e.g., foruse in motion compensation deinterlacing) and equation 20 (e.g., for usein motion adaptive deinterlacing) as follows:

τ=(3*max(|A−B|,|B−C|,|C−D|))/8  Eq. (19)

τ=(max(|A−B|,|B−C|,|C−D|))/4  Eq. (20)

Referring to FIG. 15, a diagram of multiple fields used in an examplestatic check of two samples is shown. Many overlapping operations mayexist in processing two adjacent samples and/or in different stages ofprocessing a single sample. The overlapping operations provideopportunities to reduce an overall computational complexity for thedeinterlacing process. For example, assume that the interpolated samplelocation X has been processed for an N×M static check in a first region182. To perform the N×M static check for a next interpolated samplelocation (e.g., Y) in a second region 184 (overlapping the first region182), the static check operation may perform the following checks:

-   -   (1) CURR(M)≅PREV_SAME(M)    -   (2) CURR(N)≅PREV_SAME(N)    -   (3)        |CURR(C)−PREV_OPP(C)|+|CURR(F)−PREV_OPP(C)|<p3*|CURR(B)−CURR(E)|+p4*|CURR(C)−CURR(F)|+p3*|CURR(M)−CURR(F)|,        where p3 and p4 are the programmable parameters described        before.

The earlier static check (for location X) from the first region 182 maybe saved and then partially reused for location Y in the second region184.

Referring to FIG. 16, a diagram of an example horizontal stationary edgewindow 190 is shown. Assuming that the location X has been processed, tocheck the next location Y relative to a horizontal stationary edge, anew calculation for |K−L| may be performed and the calculated results(i) for location X and (ii) among the six variations (FIG. 6) may bereused.

Referring to FIG. 17, a diagram of an example edge detection window 192is shown. Assuming that the location X has already been processed,calculating the score d(45) for the next location Y generally involvesgenerating an additional value for |X−N|. The values for |U−F|, |M−N|,|C−D| and |B−G| may have been calculated and stored earlier for thelocation X and thus may be reused for the location Y.

Referring to FIG. 18, a diagram of an example small aperture detectionwindow 194 is shown. For each sample (i,j), a minimum value (e.g., n(j))and a maximum value (e.g., m(j)) may be calculated per equations 21 and22 as follows:

m(j)=max([j−2],[j−1],[j],[j+1],[j+2])  Eq. (21)

n(j)=min([j−2],[j−1],[i],[j+1],[j+2])  Eq. (22)

The values m(j−2), m(j−1), m(j), m(j+1), m(j+2) and n(j−2), n(j−−1),n(j), n(j+1) and n(j+2) may then be stored for later use withinterpolated sample Y. To calculate the interpolated sample X, a checkfor line i and line i+1 may be made for conditions defined by equations23 and 24 as follows:

maxL>>n[j]<<maxR  Eq. (23)

minL<<m[j]>>minR  Eq. (24)

If either equation 23 or equation 24 is true for both the upper line iand the lower line i+1, directional averaging may be disabled for theinterpolated sample X.

Referring to FIG. 19, a flow diagram of an example iterativedeinterlacing method 200 is shown. The method 200 generally comprise astep (or block) 202, a step (or block) 204, a step (or block) 206, astep (or block) 208, a step (or block) 210, a step (or block) 212 and astep (or block) 214. Iterative deinterlacing generally includes two ormore passes through the interpolated samples to increase a quality ofthe interpolations. When processing the current field CURR to constructthe deinterlaced frame 100, a single-pass method may estimate themissing interpolated samples (e.g., field pixels) in raster order tosimplify an implementation in hardware. However, some interpolatedsamples may be estimated to have a certain characteristic with a highconfidence (e.g., to belong to an edge with a particular directionality)while other interpolated samples within the same spatial region may onlybe estimated with a lesser confidence. Therefore, one or more additionalpasses over the estimations may be performed using the decisions madefor the high confidence interpolated samples to influence the decisionsmade for the lower confidence neighboring interpolated samples. Multiplepass calculations may achieve a higher deinterlaced picture quality byexploiting the piecewise spatial continuity of the video (e.g., amajority of real video is generally composed of objects spanning morethan a single pixel).

The step 202 may be operational to calculate first-pass interpolatedsamples for the interpolated field OUT. The step 204 may be operationalto generate a confidence level for each of the first-pass interpolatedsamples generated by the step 202. High confidence first-pass samplesmay be marked (e.g., a first state) and low confidence first-passsamples may be not marked (e.g., a second state). Second-passinterpolated samples may be calculated in the step 206 for each of thelow confidence first-pass interpolated samples using information fromthe neighboring high confidence first-pass interpolated samplesidentified by step 204. The second-pass interpolated samples may then besubstituted for the low confidence first-pass interpolated samples inthe step 208 thus improving the interpolated field OUT.

A third pass for the interpolated field OUT may begin by generatingconfidence levels for marking/not marking the second-pass interpolatedsamples in the step 210. The step 212 may then calculate third-passinterpolate samples for each of the low confidence second-passinterpolated samples using information from the neighboring highconfidence first-pass and high confidence second-pass interpolatedsamples. The low confidence second-pass interpolated samples aregenerally replaced by the third-pass interpolated samples. The method200 may be continued with additional passes.

The (i) directional filtering and estimation blocks and the (ii)pixel-level switching and blending mechanisms of the deinterlacingmethods 102 and 200 may be combined with spatial filtering and used toprovide improved low-cost upsampling of still images. The (i)directional filtering and estimation blocks and (ii) the pixel-level andpicture-level switching and blending mechanisms may also be combinedwith spatial filtering and (optionally) motion-estimation from otherprevious frames and/or fields to provide improved low-cost upsampling ofprogressive and/or interlaced video to increase horizontal and/orvertical resolution (e.g., super resolution video). Experimental resultsfor the present invention generally indicate (i) that no new artifactsintroduced, (ii) a clear improvement on edges compared with priortechniques and (iii) that implementation may be hardware friendly.

The function performed by the flow diagrams of FIGS. 2 and 19 may beimplemented using a conventional general purpose digital computerprogrammed according to the teachings of the present specification, aswill be apparent to those skilled in the relevant art(s). Appropriatesoftware coding can readily be prepared by skilled programmers based onthe teachings of the present disclosure, as will also be apparent tothose skilled in the relevant art(s).

The present invention may also be implemented by the preparation ofASICs, FPGAs, or by interconnecting an appropriate network ofconventional component circuits, as is described herein, modificationsof which will be readily apparent to those skilled in the art(s).

The present invention thus may also include a computer product which maybe a storage medium including instructions which can be used to programa computer to perform a process in accordance with the presentinvention. The storage medium can include, but is not limited to, anytype of disk including floppy disk, optical disk, CD-ROM, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMS, Flash memory,magnetic or optical cards, or any type of media suitable for storingelectronic instructions.

While the invention has been particularly shown and described withreference to the preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade without departing from the spirit and scope of the invention.

1. A method for deinterlacing a picture, comprising the steps of: (A)calculating a potential sample at a location interfaced with a firstfield of said picture by temporal filtering; (B) evaluating a protectioncondition in a current region around said location after inclusion ofsaid potential sample; and (C) calculating an interpolated sample atsaid location by vertical spatial filtering said first field in responseto said protection condition indicating a significant increase in avertical activity within said current region due to said potentialsample.
 2. The method according to claim 1, further comprising the stepof: evaluating a similarity condition by comparing a plurality of firstsamples in said current region of said first field with a plurality ofsecond samples in a co-located region in a second field of said picture.3. The method according to claim 2, further comprising the step of:calculating said interpolated sample at said location by said temporalfiltering said first field with a third field of said picture where both(i) said similarity condition indicates that said current regionapproximately duplicates said co-located region and (ii) said protectedcondition indicates an insignificant change in said vertical activitywithin said current region.
 4. The method according to claim 1, furthercomprising the step of: evaluating said current region for a horizontalstationary edge.
 5. The method according to claim 4, further comprisingthe step of: calculating said interpolated sample at said location bysaid temporal filtering of said first field with a third field of saidpicture where said location is on a spatially stationary side of saidhorizontal stationary edge.
 6. The method according to claim 4, whereinsaid interpolated sample is calculated by said vertical spatialfiltering where said location is on a spatially dynamic side of saidhorizontal stationary edge.
 7. The method according to claim 4, whereinsaid interpolated sample is calculated by said vertical spatialfiltering where said horizontal stationary edge is not detected.
 8. Amethod for deinterlacing a picture, comprising the steps of: (A)calculating a potential sample at a location interfaced with a firstfield of said picture by temporal filtering; (B) evaluating a protectioncondition in a current region around said location after inclusion ofsaid potential sample; and (C) calculating an interpolated sample atsaid location by blend filtering in response to said protectioncondition indicating a significant increase in a vertical activitywithin said current region due to said potential sample.
 9. The methodaccording to claim 8, further comprising the step of: evaluating asimilarity condition by comparing a plurality of first samples in saidcurrent region of said first field with a plurality of second samples ina co-located region in a second field of said picture.
 10. The methodaccording to claim 9, further comprising the step of: calculating saidinterpolated sample at said location by said temporal filtering saidfirst field with a third field of said picture where both (i) saidsimilarity condition indicates that said current region approximatelyduplicates said co-located region and (ii) said protected conditionindicates an insignificant change in said vertical activity within saidcurrent region.
 11. The method according to claim 8, further comprisingthe step of: evaluating said current region for a non-horizontal edge.12. The method according to claim 11, further comprising the step of:calculating said interpolated sample at said location by directionalfiltering said first field where said location is proximate saidnon-horizontal edge.
 13. The method according to claim 11, furthercomprising the step of: calculating said interpolated sample at saidlocation by vertical spatial filtering said first field where saidnon-horizontal edge is not detected.
 14. The method according to claim13, wherein said blend filtering modifies said interpolated samplecalculated by said vertical spatial filtering by blending with one of(i) a co-located sample and (ii) a motion compensated sample.
 15. Amethod for deinterlacing a picture, comprising the steps of: (A)calculating a potential sample at a location interfaced with a firstfield of said picture by temporal filtering; (B) evaluating a protectioncondition in a current region around said location after inclusion ofsaid potential sample; and (C) calculating an interpolated sample atsaid location by directional filtering in response to said protectioncondition indicating a significant increase in a vertical activitywithin said current region due to said potential sample.
 16. The methodaccording to claim 15, further comprising the step of: evaluating asimilarity condition by comparing a plurality of first samples in saidcurrent region of said first field with a plurality of second samples ina co-located region in a second field of said picture.
 17. The methodaccording to claim 16, further comprising the step of: calculating saidinterpolated sample at said location by said temporal filtering saidfirst field with a third field of said picture where both (i) saidsimilarity condition indicates that said current region approximatelyduplicates said co-located region and (ii) said protected conditionindicates an insignificant change in said vertical activity within saidcurrent region.
 18. The method according to claim 15, further comprisingthe step of: evaluating said current region for a horizontal stationaryedge.
 19. The method according to claim 18, wherein said directionalfiltering is in further response to said horizontal edge not beingdetected.
 20. The method according to claim 18, further comprising thestep of: evaluating said current region for a non-horizontal edge,wherein said directional filtering is performed along saidnon-horizontal edge.